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ABSTRACT:  This  project  funded  the  creation  of  the  world’s  largest  online  reference  collection  of 
marine  animal  sounds  at  the  Macaulay  Library  (Cornell  Lab  of  Ornithology).  Over  a  6  year  period, 
5700  audio  clips  (1200  hours)  of  marine  mammal  and  fish  sound  recordings  were  digitized, 
documented  in  an  extensive  metadata  base,  and  made  available  for  free  online  playback.  Users  can 
search  the  collection  using  a  variety  of  criteria  and  either  play  back  the  clips  directly,  or  alternatively 
play  them  back  while  watching  customizable  and  real-time  waveforms,  spectrograms,  and  power 
spectra  Usage  of  the  collection  has  grown  7-fold  in  the  last  two  years  and  serves  the  needs  of 
researchers,  military  staff,  educators,  K-12  and  college  students,  conservation  and  wildlife  programs, 
the  arts,  museums,  zoos,  and  aquaria,  publishers,  nature  outreach  industries,  and  both  public  and 
commercial  media.  A  suite  of  30  standardized  sound  measures  were  developed  during  the  project  and 
are  now  being  tested  for  robustness  as  features  extractable  from  annotated  clips  in  the  collection. 
Lxtracted  features  will  be  used  as  additional  search  criteria,  data  for  automatic  species  identification, 
and  ecological  comparisons  of  sound  communication  in  diverse  habitats.  The  long-term  future  of  this 
collection  is  guaranteed  by  the  Macaulay  Library's  sustainability  plan  and  its  central  role  as  the  world's 
predominant  museum  of  animal  behavior  (http:/Avww.birds. Cornell. cdii/macaulavlibrarv/). 

LONG-TERM  GOALS 

The  Macaulay  Library  is  the  world's  largest  archive  of  animal  sounds  and  was  selected  by  the  Office 
of  Naval  Research  in  2002  as  a  major  repository  for  the  deposition,  digital  archival,  review,  and 
retrieval  of  the  many  recordings  of  marine  animals  made  over  the  last  half-century.  Our  long-term 
goals  have  been  to  build  as  synoptic  a  collection  a  possible,  log  full  documentation  of  all  acoustic 
specimens,  provide  the  online  tools  necessary  for  effective  search,  retrieval,  and  distribution,  and 
promote  usage  of  the  collection  by  the  worldwide  marine  bioacoustics  community  and  all  other  users 
of  rich  media  recordings  of  animal  behavior. 

OBJECTIVES 

This  project's  goals  were  to:  1 )  assemble  a  system  for  processing  aging  analog  sound  recordings  of 
marine  animals  into  high  resolution  deep  archive  storage  specimens  and  lower  resolution  files  that 
could  be  distributed,  studied,  analyzed,  and  compared  by  online  users;  2)  prioritize  archival  to  obtain 
first  broad  taxonomic  coverage  and  then  fill  in  missing  behavioral,  geographic,  age,  and  cultural 
variants;  3)  create  and  implement  a  relational  database  for  the  recordings  that  fulfills  the  metadata 
needs  of  diverse  clients  including  researchers,  educators,  conservation  biologists,  the  military,  and  the 
media;  and  4)  create  software  that:  i)  integrates  the  sound  recording  archival  system  for  maximal 
efficiency;  ii)  allows  for  automatic  extraction  of  sound  parameters  from  recordings;  iii)  permits 
metadata  annotation  from  remote  locations;  iv)  enables  users  to  search  the  database,  preview  archived 


samples,  perform  sound  analyses  online,  and  download  copies  of  the  sounds  in  any  of  a  variety  of 
formats;  v)  provides  resources  for  identification  of  unknown  marine  sounds;  vi)  supports  large  scale 
comparisons  of  sound  structures;  and  vii)  provides  sounds  for  playback  in  research  and  wildlife 
management  applications. 

APPROACH 

The  first  years  of  this  project  were  devoted  to  assembling  the  relevant  hardware,  writing  the  core 
software,  and  hiring  and  training  personnel.  The  software  tools  envisioned  included  an  online  playback 
engine  that  provided  real  time  spectrograph ic  and  waveform  visualization  of  the  sounds  being  played. 
Once  sufficient  structure  was  in  place,  we  made  a  major  effort  to  advertise  the  marine  archive  at 
national  and  international  meetings,  and  to  work  with  a  long  list  of  known  marine  recordists  to  prepare 
their  recordings  for  archival.  Recent  efforts  have  focused  on  refinement  of  the  geographic  and 
taxonomic  tools  available  during  archival  and  search,  and  creation  of  a  set  of  30  acoustic  measures  that 
can  be  extracted  from  annotated  segments  within  recordings  to  compare  with  unknown  sounds,  with 
similar  sounds  by  related  species,  or  with  sounds  produced  by  the  same  animals  in  different  habitats. 
This  will  allow  for  a  more  quantitative  use  of  the  collection  and  provide  a  reference  collection  for 
researchers  designing  automatic  and  batch  detection  tools.  We  also  currently  working  on  an  online 
registry  tool  that  will  allow  registered  users  to  select  and  download  segments  of  recordings  as  well  as 
play  them  online. 

WORK  COMPLETED 

The  archival  and  server  hardware  has  just  been  completely  upgraded  to  the  newest  Sound  Blade  and 
Apple  SAN  systems  and  should  be  sufficient  for  input  and  output  functions  for  the  next  5+  years.  A 
recent  endowment  (S5M)  made  to  the  Library  provides  a  buffer  for  necessary  repairs,  upgrades,  and 
expansions  of  the  storage  capacity  of  the  system.  The  marine  collection  is  not  only  operationally 
functional  but  it  is  also  fiscally  sustainable  at  current  levels. 

Major  efforts  were  made  to  build  a  comprehensive  marine  animal  sound  collection  from  scratch.  Most 
of  the  marine  bioacoustics  community  were  extremely  supportive  and  the  influx  of  old  and  new  audio 
recordings  has  kept  3  audio  archivists  busy  full  time  during  the  life  of  this  award.  The  collection  now 
includes  most  of  the  classic  recordings  that  were  critical  to  the  field  (e.g.  Fish,  Mowbray,  Myrbcrg  for 
fish  and  Perkins,  Ljungblad,  Steiner,  Norris,  Marshall,  Poulter,  Winn,  Levenson,  Thomas,  Miller, 
Terhune,  etc.),  and  that  has  in  turn  attracted  continuing  large  numbers  of  contributions  by  more  recent 
recordists.  Regular  reports  on  the  progress  of  the  marine  collection  and  solicitations  for  new 
contributions  were  provided  at  meetings  of  ESMF,  LCOUS,  and  most  recently,  a  meeting  for  OB1S- 
SeaMap  at  Duke  in  April  2008. 

A  complex  Oracle  relational  data  model  was  constructed  and  has  been  used  during  archival  of  all 
sounds  in  the  marine  collection.  Online  tools  allow  users  to  search  using  a  wide  variety  of  criteria,  see 
locations  of  archived  recordings  using  the  well-known  Google  map  format,  invoke  a  taxonomy  tool  to 
resolve  common  and  scientific  names,  and  recover  extensive  data  on  each  recording.  Sound  files  could 
be  played  back  for  free  using  a  REAL  plugin  to  the  user's  browser.  Because  REAL  has  become 
increasingly  unpopular  with  the  advent  of  the  ubiquitous  FLASH  (Adobe)  player  plugin,  the  Library 
converted  all  of  its  audio  files  from  REAL,  formats  to  ones  that  can  play  in  FLASH  during  spring  of 
2008.  Nearly  all  browsers  now  come  with  the  FLASH  plugin,  and  this  change  makes  audio  playback 
on  the  Macaulay  Library  site  extremely  simple  for  most  users.  This  same  player  will  also  play  back 
videos  archived  in  the  Macaulay  Library.  A  second  playback  option  available  on  the  website  provides 


visualization  of  played  back  audio.  This  tool  provides  real  time  waveform,  spectrogram,  and  power 
spectrum  views  with  settings  that  are  customizable  by  the  user.  The  original  version  of  the 
visualization  tool  was  relied  on  a  combination  of  Apple  QuickTime  and  FLASH  technologies  and  was 
posted  in  2006  where  it  won  second  prize  in  the  annual  NSF/AAAS  Technology  Challenge  for 
Interactive  Media.  In  late  2007,  Apple  unexpectedly  blocked  the  transfer  of  data  between  QuickTime 
and  FLASH  due  to  security  problems.  This  "broke"  the  existing  visualizer.  We  worked  hard 
throughout  the  winter  of  2008  to  fix  the  visualizing  tool,  and  it  is  now  up  and  running  in  a  new  form  on 
the  website.  Users  again  have  the  choice  of  simple  playback  of  audio  files  in  the  archive  or  playback 
with  associated  waveform  and  spectrogram  visualization. 

Thirty  acoustic  measures  were  derived  in  collaboration  with  Dr.  David  Mellinger  that  were  deemed 
highly  suitable  for  comparison  of  marine  animal  sounds,  provision  of  data  for  automatic  sound 
identifiers,  and  identification  of  unknown  sounds.  These  were  formulated  at  Matlab  routines  and 
posted  on  a  website  (http://mlsouiee.ornith.cornell.edu/ethoclataTeatures/)  to  solicit  trial  and  input  by 
colleagues  in  the  marine  bioacoustics  community.  Unfortunately,  implementation  of  feature  extraction 
depended  critically  on  use  of  the  online  visualization  tool.  When  Apple  broke  the  viewer,  wc  were 
unable  to  continue  development  of  the  feature  extraction  tools,  as  well  as  tools  for  expert  annotations 
of  archived  clips  and  registered  user  downloads  of  selected  segments  within  clips.  All  of  these  parts  of 
the  project  are  thus  behind  schedule,  but  are  now  being  pursued  with  funds  from  another  source. 


RESULTS 


As  of  this  report,  the  Macaulay  Library  has  archived  over  5700  acoustic  recordings  (over  1200  hours) 
of  marine  animals.  Wc  now  have  coverage  of  67%  of  Pinniped  species,  75%  of  Mysticctes,  46%  of 
Odontocetes,  and  2  of  the  5  Sircnians.  The  fish  sound  collection  has  also  flourished.  The  Macaulay 
Library  currently  has  527  sound  clips  sampled  from  143  species  of  fish  representing  93  genera  and  45 
families.  Because  96%  of  the  fish  recordings  are  from  the  Western  Atlantic  and  Caribbean,  the  Library 
is  currently  seeking  funds  to  build  a  complementary  collection  of  sounds  of  eastern  Pacific  species. 
Wherever  possible,  wc  have  prioritized  archival  to  provide  examples  of  different  classes  of  vocal 
behavior  within  a  taxon.  Online  usage  of  the  marine  sound  collection  has  increased  dramatically  from 
about  500  playbacks/month  in  early  2006  to  over  3500  playbacks/  month  currently.  The  upsurge  in 
usage  was  concurrent  both  with  implementation  of  the  online/spectrogram  player  and  concentrated 
efforts  by  Macaulay  Library  staff  to  promote  the  collection  at  national  and  international  meetings. 

IMPACT/APPLICATIONS 

The  new  search/retrieval,  playback,  and  visualization  tools  have  greatly  facilitated  access  to  the  marine 
collection.  Urgent  demands  by  television  and  radio  programs  for  exemplars  of  a  particular  species' 
sounds  can  now  be  filled  quickly  and  remotely.  Public  institutions  such  as  zoos  and  museums  can 
select  online  those  sounds  that  they  deem  arc  appropriate  for  new  exhibits.  Similarly,  the  Macaulay 
Library  has  long  been  a  source  for  reference  sounds  used  in  bioacoustic  censusing  by  wildlife 
management  programs  worldwide.  The  new  search  and  preview  tools  greatly  simply  the  task  of  finding 
suitable  recordings  for  these  functions  and  allow  these  users  to  customize  which  materials  the)  receive. 
The  growing  industry  in  popular  usage  of  natural  sounds  frequently  comes  to  the  Macaulay  Library  for 
source  materials.  There  is  widespread  interest  in  educational  and  conservation  institutions  for  access 
to  marine  animal  sounds.  These  are  now  available  online  and  can  be  played  for  free  in  any  venue. 

In  2006,  the  Macaulay  Library  began  a  program  (funded  by  the  NSDL  program  within  NSF)  to  use 
animal  behavior  as  a  springboard/entry  point  for  teaching  sciences  other  than  biology.  The  first 


application  has  been  to  help  teach  physics  in  the  K-12  age  groups.  Marine  sounds  and  marine  animals 
are  very  appealing  to  school  children,  and  by  starting  a  physics  course  at  this  level  with  examples  and 
discussions  of  marine  animal  vocal  behavior,  the  students  are  more  easily  attracted  to  the  topics.  This 
approach  has  proved  very  successful  in  pilot  programs  that  are  being  currently  tested  nationwide. 

TRANSITIONS 

Macaulay  Library  assets  are  widely  used  by  commercial  entities,  museums,  zoos,  aquaria,  science 
centers,  education,  and  the  media.  While  the  original  goal  of  the  visualization,  playback,  and 
search/retrieval  tools  was  to  facilitate  research,  we  are  now  finding  that  they  have  also  increased  access 
and  usage  of  the  archive  by  these  many  other  users.  A  number  of  commercial  products  using  Macaulay 
Library  assets  have  been  extremely  popular  including  plush  toys,  books  and  calendars  that  include  tiny 
sound  players,  and  a  recent  PDA  bird  field  guide. 

RELATED  PROJECTS 

This  project  has  been  complemented  by  a  concurrent  grant  from  the  NOPP  program  (N000 14-04- 1  - 
0663),  two  NSF  awards  as  part  of  the  National  Science  Digital  Library  program  (NSF  DUE  -0332872 
and  DUE-0532786),  and  third  NSF  award  funding  the  development  of  a  general  data  model  for  animal 
behavior  (NSF  1BN-0337507). 

HONORS/AWARDS/PRIZES 

The  Macaulay  Library’s  online  sound  and  video  player  received  second  place  for  “Interactive  Media" 
in  the  2006  NSF/AAAS  “Science  and  Engineering  Visualization  Challenge"  contest.  Details  on  this 
prize  can  be  found  online  at: 

http:/Avww. sciencemaa.org/sciext/vis20Q6/show/slidel. 3  .dtl 


